Streaming media systems transmit media data, such as video and audio data, in data packets over wired and/or wireless networks. Due to constraints on available bandwidth, for example, some data packets transmitted over a network can experience delays along the way, perhaps arriving late at a destination node. Also, some data packets may be lost along the way.
The effects of late or lost data packets may be exacerbated for video data that are predictively encoded (compressed). Predictive encoding introduces dependencies in the encoded data that improve the amount of compression but can also result in error propagation in the event of data packet loss or late arrival.
Video frames are preferably received before their display (playback) times. With predictive encoding, the decoding of a frame of data may rely on the information in another frame, and therefore some frames need to be received earlier than their display time so that they can be used to decode other frames. For example, with MPEG (Moving Pictures Experts Group) encoding, a P-frame is predicted from a preceding P-frame or I-frame, and a B-frame is predicted from two P-frames or an I-frame and P-frame. This method of encoding is used in the MPEG-1/2/4 standards, the International Telecommunication Union (ITU) H.261/2/3/4 standards, and various non-standardized compression algorithms.
Encoded video frames that do not arrive or that arrive late at the decoder (e.g., a client or destination node) will not only miss their respective display deadlines, but they may also prevent a number of other, subsequent frames from being displayed properly. If, for example, a data packet constituting an I-frame or P-frame is lost, then it might not be possible to decode and display a subsequent P-frame or B-frame. In the case in which a P-frame cannot be decoded because a preceding I-frame or P-frame is absent, other subsequent frames dependent on that P-frame may not be decoded. Thus, a number of frames may be prevented from being properly decoded and displayed due to a single late or missing frame, depending on the particular coding dependencies of the late or missing frame. This can affect the overall quality of the display.
Video transcoding offers one solution that allows encoded data to be adapted to accommodate available bandwidth and packet losses. However, transcoding can be complex and thus may require significant computational effort.
Conventional MPEG encoding lends itself to a natural method of prioritizing data packets for delivery. For example, I-frames are considered to be of higher priority than P-frames, and P-frames are considered to be of higher priority than B-frames. Thus, information about frame type can be used to make a decision about which data packet(s) should or should not be discarded or dropped because of transmission constraints.
However, MPEG encoding or streaming generally treats all frames of a given frame type the same. For instance, all P-frames are treated as having equal importance. Consequently, in a situation that involves a sequence of only P-frames, for example, there is no basis for choosing which data packet(s) should or should not be discarded or dropped because of transmission constraints. In such a situation, data packets to be dropped/discarded are randomly selected, without regard to the effect on the quality of the decoded video.